Temporal alignment system and method

ABSTRACT

A temporal alignment system and method for example for detecting temporal misalignment in video frames when the frames are divided for transport using a signal divider for dividing a single signal S into portions S 1  . . . S N  and using average picture level in determining whether data sets within a particular frame are misaligned.

PRIORITY CLAIM AND INCORPORATION BY REFERENCE

This application is a continuation of U.S. patent application Ser. No.16/183,640 filed Nov. 7, 2018 which will issue as U.S. Pat. No.10,499,049 which is a continuation of U.S. patent application Ser. No.15/859,708 filed Jan. 1, 2018 now U.S. Pat. No. 10,129,539 which is acontinuation of U.S. patent application Ser. No. 15/632,340 filed Jun.24, 2017 now U.S. Pat. No. 9,860,526 which is a continuation of U.S.patent application Ser. No. 15/261,895 now U.S. Pat. No. 9,706,198 filedSep. 10, 2016 which claims the benefit of U.S. Provisional PatentApplication No. 62/217,523 filed Sep. 11, 2015, entitled TemporalAlignment System and Method, which are incorporated herein by referencein their entirety and for all purposes.

BACKGROUND OF THE INVENTION

Broadcast and production signal transport requiring original signaldivision and multilink transport is in a protracted infancy. Reasonsinclude lack of proven equipment and movie and television industryreluctance to develop, implement, and use equipment capable oftransporting ultra-high definition signals such as or similar to 4Ksignals (see e.g., DCI 4K, UHDTV, UHD-1, UHD 4K, 4K).

FIELD OF INVENTION

This invention relates to the electrical and process arts. Inparticular, a system and method for enhancing temporal signal alignment.

DISCUSSION OF THE RELATED ART

While temporal signal alignment is well known in some applications,temporal signal alignment and realignment in the context of 4K imagestransported over multiple lines is not well known. Further, temporalsignal alignment systems operating in a 4K environment without the useof time stamps embedded in the signal are to the author's knowledgeunknown.

SUMMARY OF THE INVENTION

The present invention provides a temporal alignment system and method.Various embodiments may be used in connection with determining whetherframes, panes, signals, and/or data sets are aligned.

In an embodiment, a system for detecting temporal misalignment in videoframes that are divided for transport, the system comprises: a signaldivider for dividing a single signal S into N signal portions S₁ . . .S_(N); each signal portion for carrying a portion of the same videoframe; N transport links, each link for receiving a respective signal Sxfrom the signal divider and for delivering a signal Sx′ to a signalreassembler (1≤x≤N); the signal reassembler for selectively reassemblingsignal portions S₁′ . . . S_(N)′ corresponding to S₁ . . . S_(N) into asingle signal S′; and, a misalignment monitor colocated with the signalreassembler; wherein the monitor utilizes information inherent in thesignal portions S₁′ . . . S_(N)′ to detect whether temporal framemisalignment has occurred.

In some embodiments, the system above and one or more of i) wherein thesignals are Serial Digital Interface signals for transmitting 4K video,wherein N=4, further comprising a facility of the monitor for evaluatingaverage picture level for each of signal portions S₁′ . . . S_(N)′ andwherein the monitor facility for evaluating average picture level isused in determining whether a temporal frame misalignment has occurredand used in correcting a discovered temporal frame misalignment, iv)further comprising a facility of the monitor for evaluating averagepicture level for each of signal portions S₁′ . . . S_(N)′ and whereinchanges in average picture level caused by one or more scene cuts areused in determining whether a temporal frame misalignment has occurred,v) further comprising a facility of the monitor for evaluating averagepicture level for each of signal portions S₁′ . . . S_(N)′ and whereintemporal alignment of changes in average picture level among the signalportions S₁′ . . . S_(N)′ is used to determine whether a temporal framemisalignment has occurred, and vi) further comprising a facility of themonitor for evaluating average picture level for each of signal portionsS₁′ . . . S_(N)′ and wherein temporal alignment of discontinuities inaverage picture level among the signal portions S₁′ . . . S_(N)′ is usedto determine whether temporal frame misalignment has occurred.

In an embodiment a video frame temporal misalignment detection methodcomprising the steps of: providing a 4K video camera for acquiringimages, each image contained in a video frame that is divisible into 4panes; from one or more camera outputs, deriving 4 SDI signalscorresponding to respective video frame panes; the SDI signalstransporting the video frames over respective links such that duringtransport a pane of a particular frame is temporally misaligned andappears as a pane of another frame; providing a misalignment monitorwith a facility for monitoring average picture level of each of the SDIsignals; and, determining a temporal misalignment event has occurredwhen average picture level changes fail to occur simultaneously.

In some embodiments, the method above and one or more of the steps i)wherein the monitor is configured to detect changes in average picturelevel arising from scene cuts, wherein each pane of a frame is a onequarter slice of the image, wherein each pane of a frame reproduces theimage at a reduced resolution, and providing a frame reassembler coupledto the links and the monitor and wherein upon discovery of a video frametemporal misalignment event the monitor causes the reassembler tocorrect the misalignment by time shifting reassembler data received fromat least one of the derived SDI signals.

In an embodiment, method for finding misaligned data sets comprising thesteps of: dividing a 4K video signal for transport along 4 SDI links;transporting the 4K video signal along the SDI links; near a transportterminus, comparing temporal alignment of an average picture leveldiscontinuity in each SDI link; and, finding a temporal correction isneeded when the discontinuities are not substantially temporallyaligned.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described with reference to the accompanyingfigures. The figures, incorporated herein and forming part of thespecification, illustrate the present invention and, together with thedescription, further serve to explain the principles of the inventionand to enable a person skilled in the relevant art to make and use theinvention.

FIG. 1 shows a multi-link signal transport system of the presentinvention.

FIG. 2 shows a source with signal division for use with the system ofFIG. 1.

FIG. 3A shows a 4K image.

FIG. 3B shows a square division quad split of the image of FIG. 3A.

FIG. 4A shows a 4K image.

FIG. 4B shows an interleaved or two sample interleave of the image ofFIG. 4A.

FIG. 5 shows an exemplary multi-link signal transport system withtemporal alignment monitoring.

FIG. 6A illustrates temporally aligned signal transport 600A.

FIG. 6B shows an exemplary inherent temporal alignment assessmentmethod.

FIG. 7A illustrates temporally misaligned signal transport 700A.

FIG. 7B shows an exemplary inherent temporal misalignment assessmentmethod.

FIG. 8 shows another embodiment of the multi-link signal transportsystem with temporal alignment monitoring of FIG. 5.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The disclosure provided in the following pages describes examples ofsome embodiments of the invention. The designs, figures, anddescriptions are non-limiting examples of certain embodiments of theinvention. For example, other embodiments of the disclosed device may ormay not include the features described herein. Moreover, disclosedadvantages and benefits may apply to only certain embodiments of theinvention and should not be used to limit the disclosed inventions.

Where parts are connected, descriptions herein using the words“coupled,” “connected,” or “interconnected” refer to either direct orindirect connections. Direct connections provide for a first partconnected directly to a second part, for example A connected directly toB. Indirect connections provide for a first part connected indirectly toa second part, for example A connected indirectly to C via B.

FIG. 1 shows a multi-link signal transport system 100. As seen, atransport block 112 interconnects a source block 102 and a destinationblock 122.

In the source block 102, a signal such as an image signal is dividedamong four signal outputs s1-s4. In an embodiment, the source blockincludes a video camera that i) acquires a 4K or similar image andprovides multiple output signals s1-s4 that may be combined to reproducethe acquired 4K image.

Signals such as image signals transported from a source 102 to adestination 122, which may be and/or include a reassembler, can bedivided at the source, transported via links, and reassembled at thedestination. For example, where a 4K image is transported 112, it may betransported via multiple HD-SDI links. In particular, four (4) HD-SDIlinks 114 at 3 Gb/s might be used where each link carries one quarter(¼) of the 4K image information. As shown, signals s1-s4 from the source102 are transported via links 114 and arrive at the destination 122 assignals s1′-s4′.

Notably, to the extent that signals s1-s4 are transported 112 withtemporal and content fidelity, then signals s1′-s4′ may be reassembledto reproduce a 4K image that was acquired by the camera. However, it isnot always the case that fidelity is maintained. In particular, loss oftemporal fidelity may occur due, for example, to varying signal transittimes among the transport links 114.

FIG. 2 shows a source with signal division 200. Where content such as 4Kimagery is captured via a device such as a digital video camera, thecontent may be readied for transport by multiple links as describedabove. For example, imagery captured is a source capture block 202results in a signal “s” that is fed 203 to a source outputs block 212.The four square checkerboard icon with all four blocks marked indicatesa full bandwidth 4K signal.

In the source outputs block 212, the 4K signal is repackaged fortransport by four links. At a first output 213, a signal s1 provides ¼of the 4K bandwidth and carries ¼ of the 4K content as indicated by thetop left block of the checkerboard icon. At a second output 215, asignal s2 provides ¼ of the 4K bandwidth and carries ¼ of the 4K contentas indicated by the top right block of the checkerboard icon. At a thirdoutput 217, a signal s3 provides ¼ of the 4K bandwidth and carries ¼ ofthe 4K content as indicated by the bottom right block of thecheckerboard icon. At a fourth output 219, a signal s4 provides ¼ of the4K bandwidth and carries ¼ of the 4K content as indicated by the bottomleft block of the checkerboard icon.

Signals such as images and 4K images in 4K camera video can be carriedby multiple links as described above. FIGS. 3A-B, 4A-B illustrate twotypes of signal or image division enabling multilink transport.

FIG. 3A shows a 4K image 300A and FIG. 3B shows a square division quadsplit 300B of the image of FIG. 3A. In particular, the image of FIG. 3Ais divided into four image portions, each image portion reproducing orindicating a ¼ share of the original 4K image. Pane 1 is upper left,pane 2 is upper right, pane 4 is lower right, and pane 3 is lower left.As skilled artisans will appreciate, reproduction of the original 4Kimage from a group of 4 panes representing a single frame requirestemporal alignment of the panes.

FIG. 4A shows a 4K image 400A and FIG. 4B shows an interleaved or twosample interleave of the image 400B of FIG. 4A. As in FIGS. 3A-B, the 4Kimage is divided into 4 panes. But, here, each pane carries the wholeimage at ¼ of its original resolution. This method is referred to asinterleaving, here a 2 sample interleave. For example, pane 1 is thewhole image with 0-25% of the content, pane 2 is the whole image with26-50% of the content, pane 3 is the whole image with 51-75% of thecontent, and pane 4 is the whole image with 76-100% of the content.

FIG. 5 shows an exemplary multi-link signal transport system withtemporal alignment monitoring 500. As seen, a transport block 512interconnects a source block 502 and a destination block 522.

In the source block 502, a signal such as an image signal is dividedamong four signal outputs s1-s4. In an embodiment, the source blockincludes a video camera that i) acquires a 4K or similar image and ii)provides multiple output signals s1-s4 that may be combined to reproducethe acquired 4K image.

As described above, signals such as image signals transported from asource 502 to a destination 522 can be divided at the source,transported via links, and reassembled at the destination. Signalprocessing such as compression may occur at a first signal processor 511and signal processing such as decompression may occur at a second signalprocessor 513 (e.g., JPEG 2000, H264). In some embodiments, thecompression and decompression method provides for a lossless datatransfer.

Signals s1-s4 from the source 502 are transported via links 114 andarrive at the destination 522 as signals s1′-s4′. Notably, to the extentthat signals s1-s4 are transported 512 with temporal and contentfidelity, then signals s1′-s4′ may be combined to reproduce a 4K imagethat is a replica or near replica of the image acquired by the camera.However, it is not always the case that temporal fidelity is maintained.In particular, loss of temporal fidelity may occur due, for example, tovarying signal transit times among the transport links 114.

As shown, the transported signals s1′-s4′ are inputs to a monitor ormonitoring block 532. The monitoring block provides a means fordetermining temporal alignment and/or temporal misalignment of signalss1′-s4′. In some embodiments, the destination block 522 and monitorblock may be collocated and/or packaged in a common main (as in a rack)or sub (as in a chassis enclosure) housing. In some embodiments thepackaged destination 522 and monitor 532 blocks may be referred to as areceiver 542.

FIG. 6A illustrates temporally aligned signal transport 600A. As seen,signals s1-s4 load frames 1, 2 for transport 610. Each frame is dividedfor transport along four links 614.

At the transport input, frame 1 includes four temporally aligned datasets resulting from dividing a first signal such as a 4K signal. Frame 2includes 4 temporally aligned data sets resulting from dividing a secondsignal such as a 4K signal.

At the transport output, frame 1′ includes the four temporally aligneddata sets that were input as frame 1 and frame 2′ includes the fourtemporally aligned data sets that were input as frame 2.

Methods of monitoring the temporal alignment illustrated in FIG. 6Ainclude time stamp methods where information added, i.e., the “timestamp,” to the content being transported provides a basis for assessingtemporal alignment.

Another temporal alignment assessment method does not require thatinformation be added. Rather, information inherent in the contentprovides a basis for assessing temporal alignment. One or both ofinherent and non-inherent assessment methods may be used in the temporalalignment monitor 532 of FIG. 5.

Corresponding to FIG. 6A, FIG. 6B shows an exemplary inherent temporalalignment assessment method 600B. For each of four links and linksignals s1-s4 values of average picture level (“APL”) are plottedagainst a common time axis. In an embodiment, average picture level isan average level of the picture signal. In an embodiment, averagepicture level is the average level of the picture signal during activescanning time integrated over a frame period. APL may be defined as apercentage of the range between blanking and reference white level. Insome embodiments, any consistently used measure of APL suffices (seee.g., IDMS 1.03 Information Display Measurements Standard (IDMS),pre-gamma APL (Type 1 APL) for gamma corrected input signal(R, G, B) andpost-gamma APL (Type 2 APL) for gamma de-corrected panel displaysignal(R′, G′, B′)).

As the plots show, APL data on each of the links (APLs1 . . . APLs4)indicates a change or discontinuity in average picture level such as anAPL change associated with a scene cut. Moreover, this change in APLoccurs simultaneously, at time t1, for all of the links.

Monitoring this information, temporal alignment of changes ordiscontinuities, provides a method of finding when data on multiplelinks representing a particular frame is not temporally aligned.Notably, this telltale sign of temporal misalignment is part of thepicture content and thus is inherent in the signals being transported.

FIG. 7A illustrates a signal transport that is temporally misaligned700A. As seen, signals s1-s4 load frames 1, 2 for transport 710. Eachframe is divided for transport along four links 714.

At the transport input, frame 1 includes four temporally aligned datasets resulting from dividing a first signal such as a 4K signal. Frame 2includes 4 temporally aligned data sets resulting from dividing a secondsignal such as a 4K signal.

At the transport output, frame 1′ includes three temporally aligned datasets that were input as frame 1 and one temporally misaligned data set731. Frame 1′ is therefore temporally misaligned, for example becausethe frame excludes data it should include or includes input data setsfrom different input data frames. In similar fashion, frame 2′ istemporally misaligned, for example because it includes a misalignedframe 733.

Corresponding to FIG. 7A, FIG. 7B shows an assessment of temporalalignment 700B. For each of four links and link signals s1-s4 values ofaverage picture level (“APL”) are plotted against a common time axis.

As the plots show, APL data on each of the links (APLs1 . . . APLs4)indicates a change or discontinuity in average picture level such as anAPL change associated with a scene cut. As the plots show, APL data oneach of the links (APLs1 . . . APLs4) indicates a change ordiscontinuity in average picture level such as an APL change associatedwith a scene cut.

For three of the links, APLs1, APLs3, APLs4, the change in APL occurssimultaneously, at time t1 and corresponds with a first discontinuity,for example a scene cut, such as scene cut 1.

For one of the links, APLs2, the change in APL occurs at a differenttime t2 corresponding with a second discontinuity such as scene cut 2where t2−t1=dt.

Monitoring this information, temporal alignment of changes ordiscontinuities, provides a method of finding when data on multiplelinks representing a particular frame is not temporally aligned.Notably, this telltale sign of temporal misalignment is part of thepicture content and thus is inherent in the signals being transported.

FIG. 8 shows another embodiment 800 of the multi-link signal transportsystem with temporal alignment monitoring of FIG. 5.

In an embodiment, a first signal line 551 interconnecting the monitorand transport blocks 532, 512 enables the monitor block to realign datasets along links such that data sets within frames are aligned. Controlfunction capabilities include speeding or slowing transport throughputon any one or more of the links 514.

In an embodiment, a second signal line 553 interconnecting the monitorand source blocks 532, 502 enables the monitor block to realign datasets along links such that data sets within frames are aligned. Controlfunction capabilities including speeding or slowing source data ratesalong any one or more of the data source outputs.

The appendix to this application provides descriptions of similar and/orother embodiments of the present invention.

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. It will be apparent to those skilledin the art that various changes in the form and details can be madewithout departing from the spirit and scope of the invention. As such,the breadth and scope of the present invention should not be limited bythe above-described exemplary embodiments, but should be defined only inaccordance with the following claims and equivalents thereof.

What is claimed is:
 1. A method for aligning video signal portions, themethod comprising the steps of: providing a video camera for acquiring avideo signal derived from an image and having an inherent video signalfeature; transporting parts of the video signal along plural links; inplural links, comparing temporal alignment of a change of state of thefeature; and, where misalignment is found, changing the temporalalignment of one or more transported signals.
 2. The method of claim 1wherein each link transports a unique portion of the image.
 3. Themethod of claim 2 wherein each link transports a different area of theimage.
 4. The method of claim 2 wherein each link transports a similararea of the image.
 5. The method of claim 1 wherein the video signal isa Serial Digital Interface (“SDI”) signal.
 6. The method of claim 1further comprising the steps of: compressing the video signal beforetransporting the video signal along multiple links and decompressing thevideo signal after transporting the video signal.
 7. The method of claim1 wherein the video signal feature is related to average picture level(APL).
 8. The method of claim 1 wherein the change of state correspondsto a scene cut.
 9. The method of claim 1 further including the steps of:monitoring the transported parts of the video signal along multiplelinks; and, based on the monitoring, varying transport rates on one ormore links to change the temporal alignment of one or more transportedsignals.
 10. The method of claim 1 further including the steps of:monitoring a video signal source; and, based on the monitoring, varyingone or more source data rates to change the temporal alignment of one ormore transported signals.
 11. A video signal alignment apparatuscomprising: a video camera for acquiring a video signal derived from animage and having an inherent video signal feature; plural links fortransporting parts of the video signal to a destination block via asignal compression block, a transport block, and a signal decompressionblock; a monitoring block for determining at least one of video signaltemporal alignment and video signal misalignment; and, the monitoringblock for advising the transport block of misalignment; wherein inplural links the temporal alignment of a change of state of the featureis compared and where misalignment is found the temporal alignment ofone or more transported signals is changed.
 12. The alignment apparatusof claim 11 wherein each link transports a unique portion of the image.13. The alignment apparatus of claim 12 wherein each link transports adifferent area of the image.
 14. The alignment apparatus of claim 13wherein each link transports a similar area of the image.
 15. Thealignment apparatus of claim 11 wherein the video signal is a SerialDigital Interface (“SDI”) signal.
 16. The alignment apparatus of claim11 wherein the video signal feature is related to average picture level(APL).
 17. The alignment apparatus of claim 11 wherein the change ofstate corresponds to a scene cut.
 18. The alignment apparatus of claim11 further comprising changes to transport rates on one or more links tochange the temporal alignment of one or more transported signals.